{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q Networks\n",
    "前面我们介绍了强化学习中的 q-learning，我们知道对于 q-learning，我们需要使用一个 Q 表来存储我们的状态和动作，每次我们使用 agent 不断探索环境来更新 Q 表，最后我们能够根据 Q 表中的状态和动作来选择最优的策略。但是使用这种方式有一个很大的局限性，如果在现实生活中，情况就会变得非常的复杂，我们可能有成千上万个 state，甚至是无穷无尽有可能是无穷无尽的 state，对于这种情况，我们不可能将所有的 state 都用 Q 表来存储，那么我们该如何解决这个问题呢？\n",
    "\n",
    "一个非常简单的办法就是使用深度学习来解决这个问题，所以出现了一种新的网络，叫做 Deep Q Networks，将 Q learning 和 神经网络结合在了一起，对于每一个 state，我们都可以使用神经网络来计算对应动作的值，就不在需要建立一张表格，而且网络更新比表格更新更有效率，获取结果也更加高效。\n",
    "\n",
    "![](https://ws4.sinaimg.cn/large/006tKfTcgy1fni66at6jbj30xo0g1jut.jpg)\n",
    "\n",
    "下面我们使用 open ai gym 环境中的 CartPole 来尝试实现一个简单的 DQN。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "import numpy as np\n",
    "import gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一些超参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "lr = 0.01\n",
    "epsilon = 0.9\n",
    "gamma = 0.9\n",
    "target_replace_iter = 100\n",
    "memory_capacity = 2000\n",
    "env = gym.make('CartPole-v0')\n",
    "env = env.unwrapped\n",
    "n_actions = env.action_space.n\n",
    "n_states = env.observation_space.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of actions are: 2\n",
      "number of states are: 4\n"
     ]
    }
   ],
   "source": [
    "print('number of actions are: {}'.format(n_actions))\n",
    "print('number of states are: {}'.format(n_states))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用 gym 自带的环境，关于 CartPole 的一些参数，我们同样可以像上一节课 MountainCar 一样找到，[地址]( https://github.com/openai/gym/wiki/CartPole-v0) 在这里，通过这里可以看到，这个环境的观测的 4 个值分别是板的位置，板的速度，杆的角度以及杆顶端的速度，我们能够采取的动作就是 2 个，分别是向左移动杆和向右移动杆。\n",
    "\n",
    "本质上，我们其实可以不用关心状态到底是什么，这些状态都可以作为神经网络的输入，输出就是每个动作的 value，可以让神经网络自己学会建立一个类似 Q 表的东西。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class q_net(nn.Module):\n",
    "    def __init__(self, hidden=50):\n",
    "        super(q_net, self).__init__()\n",
    "        self.fc = nn.Sequential(\n",
    "            nn.Linear(n_states, hidden),\n",
    "            nn.ReLU(True),\n",
    "            nn.Linear(hidden, n_actions)\n",
    "        )\n",
    "        \n",
    "        nn.init.normal(self.fc[0].weight, std=0.1) # 使用标准差是 0.1 的正态分布初始化\n",
    "        nn.init.normal(self.fc[2].weight, std=0.1) # 使用标准差是 0.1 的正态分布初始化\n",
    "        \n",
    "    def forward(self, x):\n",
    "        actions_value = self.fc(x)\n",
    "        return actions_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来我们定义一个 DQN 的过程作为一个类，在这个类中，我们会定义 dqn 的学习过程，dqn 的更新过程和之前讲个 q learning 的过程是很相同的\n",
    "\n",
    "![](https://ws1.sinaimg.cn/large/006tNc79ly1fnhl68e9spj30ek0biwfw.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图中有两个网络，第一个网络叫做 action-value function Q，还有一个网络叫做 target action-value $\\hat{Q}$，第一个网络可以称为 eval net，第二个网络叫做 target net，eval net 每一步都更新参数，而 target net 每过 C 步才更新一次，更新地方式是直接将 eval net 的参数复制到 target net 上，为什么要学习两个网络而不学习一个网络呢？因为 DQN 希望每次更新能够有一定的时间去探索周围的环境而得到更多的信息再来更新，在游戏中的过程是一个时间序列，所以得到的信息是有着时间依赖关系，每次都直接使用序列信息进行更新并不能得到好的结果，所以我们就采用滞后更新的方式，每隔 C 步才真正更新一次参数。\n",
    "\n",
    "DQN 的学习过程也非常简单，我们使用 eval net 作为估计动作的 value，使用 target net 得到动作实际的 value，我们希望估计的 value 能够等于实际的 value，所以可以使用 mse 来作为 loss 函数就可以了。\n",
    "\n",
    "下面我们来实现整个过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DQN(object):\n",
    "    def __init__(self):\n",
    "        self.eval_net, self.target_net = q_net(), q_net()\n",
    "\n",
    "        self.learn_step_counter = 0\n",
    "        self.memory_counter = 0\n",
    "        self.memory = np.zeros((memory_capacity, n_states * 2 + 2)) # 当前的状态和动作，之后的状态和动作\n",
    "        self.optimizer = torch.optim.Adam(self.eval_net.parameters(), lr=lr)\n",
    "        self.criterion = nn.MSELoss()\n",
    " \n",
    "    def choose_action(self, s):\n",
    "        '''\n",
    "        根据输入的状态得到所有可行动作的价值估计\n",
    "        '''\n",
    "        s = Variable(torch.unsqueeze(torch.FloatTensor(s), 0))\n",
    "        # input only one sample\n",
    "        if np.random.uniform() < epsilon:  # greedy 贪婪算法\n",
    "            actions_value = self.eval_net(s)\n",
    "            action = torch.max(actions_value, 1)[1].data[0]\n",
    "        else:  # random 随机选择\n",
    "            action = np.random.randint(0, n_actions)\n",
    "        return action\n",
    "\n",
    "    def store_transition(self, s, a, r, s_):\n",
    "        transition = np.hstack((s, [a, r], s_))\n",
    "        # 用新的记忆替换旧的记忆\n",
    "        index = self.memory_counter % memory_capacity\n",
    "        self.memory[index, :] = transition\n",
    "        self.memory_counter += 1\n",
    "\n",
    "    def learn(self):\n",
    "        # target net 的参数更新\n",
    "        if self.learn_step_counter % target_replace_iter == 0:\n",
    "            self.target_net.load_state_dict(self.eval_net.state_dict())\n",
    "        self.learn_step_counter += 1\n",
    "\n",
    "        # 取样记忆中的经历\n",
    "        sample_index = np.random.choice(memory_capacity, batch_size)\n",
    "        b_memory = self.memory[sample_index, :]\n",
    "        b_s = Variable(torch.FloatTensor(b_memory[:, :n_states]))\n",
    "        b_a = Variable(\n",
    "            torch.LongTensor(b_memory[:, n_states:n_states + 1].astype(int)))\n",
    "        b_r = Variable(\n",
    "            torch.FloatTensor(b_memory[:, n_states + 1:n_states + 2]))\n",
    "        b_s_ = Variable(torch.FloatTensor(b_memory[:, -n_states:]))\n",
    "\n",
    "        # q_eval net 评估状态下动作的 value\n",
    "        q_eval = self.eval_net(b_s).gather(1, b_a)  # shape (batch, 1) 选择对应 action 的动作\n",
    "        q_next = self.target_net(\n",
    "            b_s_).detach()  # detach from graph, don't backpropagate\n",
    "        q_target = b_r + gamma * q_next.max(1)[0].view(batch_size, 1)  # shape (batch, 1)\n",
    "        loss = self.criterion(q_eval, q_target) # mse 作为 loss 函数\n",
    "        # 更新网络\n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        self.optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dqn_trainer = DQN()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "collecting experience ... \n",
      "Ep: 199 | reward: 1.720\n",
      "Ep: 200 | reward: 4.116\n",
      "Ep: 201 | reward: 2.598\n",
      "Ep: 202 | reward: 7.261\n",
      "Ep: 203 | reward: 1.570\n",
      "Ep: 204 | reward: 1.191\n",
      "Ep: 205 | reward: 2.557\n",
      "Ep: 206 | reward: 2.822\n",
      "Ep: 207 | reward: 1.788\n",
      "Ep: 208 | reward: 3.345\n",
      "Ep: 209 | reward: 2.787\n",
      "Ep: 210 | reward: 2.718\n",
      "Ep: 211 | reward: 21.747\n",
      "Ep: 212 | reward: 2.514\n",
      "Ep: 213 | reward: 1.504\n",
      "Ep: 214 | reward: 5.059\n",
      "Ep: 215 | reward: -1.003\n",
      "Ep: 216 | reward: 5.880\n",
      "Ep: 217 | reward: 25.360\n",
      "Ep: 218 | reward: 19.871\n",
      "Ep: 219 | reward: 6.934\n",
      "Ep: 220 | reward: 35.473\n",
      "Ep: 221 | reward: 46.048\n",
      "Ep: 222 | reward: 34.585\n",
      "Ep: 223 | reward: 103.504\n",
      "Ep: 224 | reward: 92.848\n",
      "Ep: 225 | reward: 61.636\n",
      "Ep: 226 | reward: 68.378\n",
      "Ep: 227 | reward: 150.926\n",
      "Ep: 228 | reward: 358.854\n",
      "Ep: 229 | reward: -303.586\n",
      "Ep: 230 | reward: 44.629\n",
      "Ep: 231 | reward: 271.936\n",
      "Ep: 232 | reward: 93.458\n",
      "Ep: 233 | reward: 203.914\n",
      "Ep: 234 | reward: 126.819\n",
      "Ep: 235 | reward: 492.525\n",
      "Ep: 236 | reward: 865.907\n",
      "Ep: 237 | reward: 457.963\n",
      "Ep: 238 | reward: 189.550\n",
      "Ep: 239 | reward: 654.743\n",
      "Ep: 240 | reward: 526.357\n",
      "Ep: 241 | reward: 262.288\n",
      "Ep: 242 | reward: 496.026\n",
      "Ep: 243 | reward: 266.828\n",
      "Ep: 244 | reward: 125.170\n",
      "Ep: 245 | reward: 1129.094\n",
      "Ep: 246 | reward: 1046.364\n",
      "Ep: 247 | reward: 3.924\n",
      "Ep: 248 | reward: 54.877\n",
      "Ep: 249 | reward: 790.436\n",
      "Ep: 250 | reward: 163.135\n",
      "Ep: 251 | reward: 439.592\n",
      "Ep: 252 | reward: 286.395\n",
      "Ep: 253 | reward: 1096.519\n",
      "Ep: 254 | reward: 198.122\n",
      "Ep: 255 | reward: 278.892\n",
      "Ep: 256 | reward: 215.559\n",
      "Ep: 257 | reward: 182.856\n",
      "Ep: 258 | reward: 700.339\n",
      "Ep: 259 | reward: 333.089\n",
      "Ep: 260 | reward: 172.095\n",
      "Ep: 261 | reward: 222.899\n",
      "Ep: 262 | reward: 770.689\n",
      "Ep: 263 | reward: 1006.111\n",
      "Ep: 264 | reward: 680.227\n",
      "Ep: 265 | reward: 313.157\n",
      "Ep: 266 | reward: 445.412\n",
      "Ep: 267 | reward: 254.287\n",
      "Ep: 268 | reward: 286.542\n",
      "Ep: 269 | reward: 137.780\n",
      "Ep: 270 | reward: 240.850\n",
      "Ep: 271 | reward: 173.953\n",
      "Ep: 272 | reward: 929.376\n",
      "Ep: 273 | reward: 1682.483\n",
      "Ep: 274 | reward: 124.281\n",
      "Ep: 275 | reward: 439.573\n",
      "Ep: 276 | reward: 388.455\n",
      "Ep: 277 | reward: 366.418\n",
      "Ep: 278 | reward: 594.223\n",
      "Ep: 279 | reward: 385.194\n",
      "Ep: 280 | reward: 702.398\n",
      "Ep: 281 | reward: 121.711\n",
      "Ep: 282 | reward: 555.598\n",
      "Ep: 283 | reward: 617.005\n",
      "Ep: 284 | reward: 1366.853\n",
      "Ep: 285 | reward: 667.864\n",
      "Ep: 286 | reward: 308.383\n",
      "Ep: 287 | reward: 332.831\n",
      "Ep: 288 | reward: 614.663\n",
      "Ep: 289 | reward: 1077.267\n",
      "Ep: 290 | reward: 385.115\n",
      "Ep: 291 | reward: 375.811\n",
      "Ep: 292 | reward: 183.127\n",
      "Ep: 293 | reward: 132.291\n",
      "Ep: 294 | reward: 161.788\n",
      "Ep: 295 | reward: 133.430\n",
      "Ep: 296 | reward: 455.821\n",
      "Ep: 297 | reward: 392.828\n",
      "Ep: 298 | reward: 172.488\n",
      "Ep: 299 | reward: 756.369\n"
     ]
    }
   ],
   "source": [
    "print('collecting experience ... ')\n",
    "all_reward = []\n",
    "for i_episode in range(300):\n",
    "    s = env.reset()\n",
    "    reward = 0\n",
    "    while True:\n",
    "#         env.render()\n",
    "        a = dqn_trainer.choose_action(s)\n",
    "\n",
    "        # 环境采取动作得到结果\n",
    "        s_, r, done, info = env.step(a)\n",
    "\n",
    "        # 修改奖励以便更快收敛\n",
    "        x, x_dot, theta, theta_dot = s_\n",
    "        r1 = (env.x_threshold - abs(x)) / env.x_threshold - 0.8\n",
    "        r2 = (env.theta_threshold_radians - abs(theta)) / env.theta_threshold_radians - 0.5\n",
    "        r = r1 + r2\n",
    "\n",
    "        dqn_trainer.store_transition(s, a, r, s_)\n",
    "\n",
    "        reward += r\n",
    "        if dqn_trainer.memory_counter > memory_capacity: # 记忆收集够开始学习\n",
    "            dqn_trainer.learn()\n",
    "            if done:\n",
    "                print('Ep: {} | reward: {:.3f}'.format(i_episode, round(reward, 3)))\n",
    "                all_reward.append(reward)\n",
    "                break\n",
    "\n",
    "        if done:\n",
    "            break\n",
    "        s = s_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x119000ba8>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztvXl4ZGd95/v51SqV1tbW+2Z3e7dp\n240xi81iCMYhGAhJ7Jsh3Mli1oRkcu8EQu6ThAzZEyZMwMEsARKGZbABh5gEcIjBAWO3wWnb7a3d\nq3qTWmqttVe9949z3qpTe5VUi6T+fZ5Hj6RTp6reqpLe7/ntYoxBURRFOb/xdXoBiqIoSudRMVAU\nRVFUDBRFURQVA0VRFAUVA0VRFAUVA0VRFAUVA0VRFAUVA0VRFAUVA0VRFAUI1HuiiHwaeD0wYYy5\nwj32JeBi95RBYMYYs0dEdgBPAc+4tz1kjHmHe59rgc8A3cB9wHtNHWXQIyMjZseOHfUuV1EU5bzn\n0UcfPWuMGa3n3LrFAGcD/1vgc/aAMeYX7M8i8lfArOf8540xe8o8zp3AHcBDOGJwM/DNWk++Y8cO\n9u3b18ByFUVRzm9E5Gi959btJjLGfA+YrvCEAvw88IUaC9sI9BtjfuhaA58D3ljvGhRFUZTW0KyY\nwQ3AGWPMc55jO0XkJyLygIjc4B7bDIx7zhl3jymKoigdpBE3UTVup9AqOAVsM8ZMuTGCr4nI5YCU\nuW/FeIGI3IHjUmLbtm1NWqqiKIpSzLItAxEJAG8GvmSPGWMSxpgp9+dHgeeBi3AsgS2eu28BTlZ6\nbGPMXcaYvcaYvaOjdcVAFEVRlCXQDDfRq4GnjTE594+IjIqI3/35AmA3cMgYcwqYF5Hr3TjDLwFf\nb8IaFEVRlGVQtxiIyBeAHwIXi8i4iPyKe9NtlAaObwT2i8h/Al8B3mGMscHndwKfBA7iWAw1M4kU\nRVGU1iKrZdLZ3r17jaaWKoqi1I+IPGqM2VvPuVqBrChKS3nixCw/OXau08tQaqBioChKS/mzf3ma\nD/3zU51ehlKDZqWWKoqilGU+niaTXR3u6PMZFQNFUVpKPJXp9BKUOlA3kaIoLSWazJBWy2DFo5aB\noigtJZbK4CvXe0BZUahloChKS4klM6QyahmsdFQMFEVpGcYYYqkM6Wy200tRaqBioChKy0hlDJms\nIa2WwYpHxUBRlJYRSzqZRMmMWgYrHRUDRVFaRsxNK1XLYOWjYqAoSsuIJtMAGjNYBagYKIrSMqxl\nkMoYVktTzPMVFQNFUVqGt/pYW1KsbFQMFEVpGdFkXgy0Cnllo2KgKErLiHnEIKUZRSuaRiadfVpE\nJkTkCc+xPxCREyLymPt1i+e294vIQRF5RkRe6zl+s3vsoIi8r3kvRVGUlUbM4ybSjKKVTSOWwWeA\nm8sc/7AxZo/7dR+AiFyGMw7zcvc+HxMRvzsX+aPA64DLgNvdcxVFWYMUWAaaUbSiqbtRnTHmeyKy\no87TbwW+aIxJAIdF5CBwnXvbQWPMIQAR+aJ77oG6V6woyqpBLYPVQzNiBu8Rkf2uG2mde2wzcNxz\nzrh7rNLxsojIHSKyT0T2TU5ONmGpiqK0k4IAsorBima5YnAncCGwBzgF/JV7vFzDWlPleFmMMXcZ\nY/YaY/aOjo4uc6mKorQbb2qpuolWNsuaZ2CMOWN/FpFPAN9wfx0HtnpO3QKcdH+udFxRlDWGWgar\nh2VZBiKy0fPrmwCbaXQvcJuIhEVkJ7AbeBh4BNgtIjtFJIQTZL53OWtQFGXl4o0ZaGrpyqZuy0BE\nvgC8AhgRkXHg94FXiMgeHFfPEeDtAMaYJ0XkyziB4TTwbmNMxn2c9wD/CviBTxtjnmzaq1EUZUUR\n16KzVUMj2US3lzn8qSrnfwj4UJnj9wH31fu8iqKsXgrdRGoZrGS0AllRlJZR6CZSy2Alo2KgKErL\niKUyBHxOEqG2sV7ZqBgoitIyYskM/d1BQLOJVjoqBoqitIxYKkNflxOa1GyilY2KgaIoLSOWzIuB\nZhOtbFQMFEVpGbFUhv4ux02klsHKRsVAUZSWUWAZaMxgRaNioChKS8hmTYFl0MlsosNnF7no977J\nocmFjq1hpaNioChKS0iknc2/L+cm6pxlcGRqkWQ6y9GpaMfWsNJRMVAUpSVEk2kAj5uoc5aBbYvh\nrYhWClExUBSlJdjq41ydQQeziaI5MUh3bA0rHRUDRVFagp1lkK8z6JwYWGHytsdQClExUBSlJdir\n8VwAuYNuopi6iWqiYqAoSkuwG3DOMuigm8haBNGEuokqoWKgKEpLiLobcHfIT8AnHbUMomoZ1ETF\nQFGUlmAzeCIhPwG/dDSAbOMXUY0ZVKRuMRCRT4vIhIg84Tn2FyLytIjsF5Gvisige3yHiMRE5DH3\n6+8897lWRB4XkYMi8hERkea+JEVRVgLWNdMd9BP0+TrajsK6rGJqGVSkEcvgM8DNRce+DVxhjLkK\neBZ4v+e2540xe9yvd3iO3wncgTMXeXeZx1QUZQ1gXTLdQdcy6GA2kbUINLW0MnWLgTHme8B00bFv\nGWPsu/sQsKXaY4jIRqDfGPNDY4wBPge8sbElK4qyGoh7YwZ+X0fbUWg2UW2aGTP4ZeCbnt93ishP\nROQBEbnBPbYZGPecM+4eK4uI3CEi+0Rk3+TkZBOXqihKq/FaBkGfdLjOIF2wJqWUpoiBiHwASAOf\ndw+dArYZY64G/hvwv0WkHygXH6j4F2KMucsYs9cYs3d0dLQZS1UUpU3EUhlCfh8B90vrDFY2geU+\ngIi8DXg9cJPr+sEYkwAS7s+PisjzwEU4loDXlbQFOLncNSiKsvKIJTN0BZ3rzYBfOlpnEM0FkDVm\nUIllWQYicjPwO8AbjDFRz/FREfG7P1+AEyg+ZIw5BcyLyPVuFtEvAV9fzhoURVmZxJIZukN+AIK+\nzloGudRStQwqUrdlICJfAF4BjIjIOPD7ONlDYeDbboboQ27m0I3AB0UkDWSAdxhjbPD5nTiZSd04\nMQZvnEFRlDVCLJUhEnK2mI5nE6mbqCZ1i4Ex5vYyhz9V4dy7gbsr3LYPuKLe51UUZXUSTWboCjqW\nQcDvWxntKJJpjDFoeVMpWoGsKEpLiKcyRHJuos62o4inMvgEsiY/dEcpRMVAUZSWEEtl6M5ZBp1z\nE6UyWVIZw1BPyFmXuorKomKgKEpL8LqJgn4fqQ4VnVkXkRUD7U9UHhUDRVFagtdN5HQt7YxlYC2B\n4Z4woG2sK6FioChKS4glvW6izjWqy4lBr2sZqJuoLCoGiuLhR4emmJiPd3oZa4JoMp2vM+hgC2u7\n+Y/0hgt+VwpRMVAUD7/6uX384b0HOr2MNUE8lc2JQaCDRWfFMQPbp0gpRMVAUVyMMSwm0nz7wBlm\no6lOL2dVk85kSWayBdlEnWpUp26i+lAxUBSXdNaQNZDMZPnG461pmZVMZ1k4DwKY9mo84mlH0bGY\nQao4gKxiUA4VA0Vx8RYj3f3oeJUzl87f3P8sb7nzBy157JWE3YBzqaWBzsUM7FpGcpbB2hfjpaBi\noCguSVcMtqzr5sfHZjg0udD05zhxLsap2bUfoI4lCy2DQCctA3fzX06dwfHpKIn02rYoVAwUxcX+\ns//ctVvxCXz1Jyea/hyxVCbXQXMt451/DG42UYdjBoORECKNVyAn0hle+z+/x5cfOd6K5a0YVAwU\nxSWRcq5ctw5187Ldo9zz4xNkm+zaiKeyJNJZ3NEfaxYbpO0K5esMOjX2MuqJX0SC/oYDyDPRFNFk\nhjNziVYsb8WgYqAoLjZmEA74+dlrNnNiJsaPDk/XuFdj2Cvmtd4sLW7dRMF8o7pUxnREBOPJDCIQ\nDviIhAMNxwxmY05m2VrPQlIxUBQX6yYKB3z81GUb6A76+daB0019DusislbIWiXnJvJYBgCZDgSR\no24ltIgQCTVuGVgxWOv1CSoGiuJiA8jhoI/ukJ91kSDz8eZuAFYM4ms8GBktDiD7nfkBncgoinl6\nJHUvwU1ka04W13hKakNiICKfFpEJEXnCc2xIRL4tIs+539e5x0VEPiIiB0Vkv4hc47nP29zzn3Nn\nKCtKx/G6iQDCQX/T3Tn2inmtB5FLUkt9zlbTiYyimKd7aiTkbziArG6i8nwGuLno2PuA+40xu4H7\n3d8BXocz+3g3cAdwJzjigTMy80XAdcDvWwFRlE5i3UShgPNvEfL7SDb5Cj6WzLrP1f5NcXox2Taf\nvd1wvRXIQEcyiryWQSS09JiBuok8GGO+BxRH1G4FPuv+/FngjZ7jnzMODwGDIrIReC3wbWPMtDHm\nHPBtSgVGUdqO9eOHXTEIB31N37QTHbIMxs9Fue5D3+E/Dk615fnyFch2BrJrGXQgo8g7ZGc5MQN1\nE9VmvTHmFID7fcw9vhnwJuaOu8cqHS9BRO4QkX0ism9ycrIJS1WUyiQzhWLgWAatchO1d1N89sw8\n6azh4MR8W57PWgb2vQz6OmcZRIvcREsOIKubaMmUmzhtqhwvPWjMXcaYvcaYvaOjo01dnKIUk7MM\ngjZm0FzLIJXJ5gKo7a5mPTYVBWByoT258vZq3OeKgLUMOiEG3iE73aFAw2IwF3djBuomqskZ1/2D\n+33CPT4ObPWctwU4WeW4onQUb2opNN8y8LqG2m0ZHD8XA2Byvk1ikMzk0krBqUCGzriJop61OAHk\nxjb1ObUM6uZewGYEvQ34uuf4L7lZRdcDs64b6V+BnxKRdW7g+KfcY4rSUawVYAPI4YC/qVfwsQIx\naO/GcnzatQzaJAY2t98S8HXOMnAmrjmxi0jITzSVaSiQfr7EDAKNnCwiXwBeAYyIyDhOVtCfAl8W\nkV8BjgE/555+H3ALcBCIAv8VwBgzLSJ/BDzinvdBY0xzyzwVZQnkU0tdyyDQXMvAW2jWdjGwlkGb\n3ETxVKFlYLOJOpJamsrQHXI+00gogDGFg3dqkc8mypDNmpzra63RkBgYY26vcNNNZc41wLsrPM6n\ngU838tyK0mpyloHfWgbNjRl4LYN2ppYaY9puGXgzeCDvJupI0Vkyk8tqsrED70jOWlgxAKdY0D7W\nWkMrkBXFJZHOEA74EHE2rmZbBl6fczstg5loioVEmp6Qn7MLyaY33ytH8WabdxO11zLIZg2xVD6b\nqDsnBvW//7OxVE7Y1rKrSMVAUVwSqWzORQQ2ZtCaAHI7LYPj5xyrYM+2QTJZw7losuXPGUtlcxsw\neN1E7bUM7Psc8QSQnfXVt6kn0hniqSwbB7uc+63hILKKgaK4JNJZQoH8BtZ0y8ArBm20DI5PO/GC\na7Y5hf7tiBvEk5lcx1KAoE0tbXM2ka02tlf2Pa6Lp17LwLqINvQ7YrCW00tVDBTFxbqJLOGAj2Qm\n2zS3SkFqaRstg2NuvCAnBm2IG0RT6dxVOECgQ0VnxUN2cm6iOudQ27TSDQOOGKibSFHOA5LpLOFg\n/l/Cppgmm+Tnjncom+j4uSjrIkF2jPQA7RGDWDKbG2wDecug3dlE9n3uLnITNWoZbBroBtRNpCjn\nBYl0NtexFPIpps3y79urVJE2i8F0lG1DEUb7wkC7xCBd4CbqVAvraFHDvJwY1Pn+zxZZBo02uVtN\nqBgoKxpjDCdmYm15LidmUOgmApoWN7BXlQPdwbYGkMfPxdgyFKEn5Kc76G+5GBhj3Nz+0myidlsG\nsaK5Ct1uzKDeKmQrBhtzYqCWgaJ0hH9/ZpIb/uzf2iIIiVRxzMDZQJpVhWwH2gx2B9tmGWSyhhPn\nYmxdF0FEGO0LtzyAnMxkyRoKsomCHWphbS0A67LqadRNFLVi0N3Q/VYjKgbKiubI1CJZAyfbIQbp\nwtTSUJMtAzuLt7872LbeRGfm4iQzWbYOOZvZaF+45ZZB8dU4eBrVtTmbKF5iGTQmBnPupDt1EylK\nh5ledHLizy22Pjc+2YaYQVfAT1eTex5Vw1YebxuKADDa2wYxKMrggXwL63bXGRTHDEJ+H36f1L2p\nz8ZS9IT89HU1lpK6GlExUFY0U64IzERTNc5cPol0pnw2UbMsA7cfTjjoa5tlYHsSbV3nikEb3ES5\nKWflLIN2xwyKhElEiDQwB3k2lmKgO0jQ7yPk96kYKEqnmF5wLYM2VM0m0lnC/nIxg+ZZBt1BP11B\nf9tiBseno4jApsG8m2gmmmqpZVJ8NQ6dyyYqJ0zdDcxBno2l6O8Oeu6nbiJF6QjWTTTdLjFooWUQ\nSzmWR7Mb4FXj+HSUjf1dudcy5qaXTi207v0szu0HCOayiTpbdAbQE65/wI21DMAJPqtloCgdYmrR\ncWnMLLbBTZTKVIgZNGcDSHgsg3a1ozh+LspWN14AtKXWoKpl0AE3Ucjvy7mp7LrqjRnMFVkGKgaK\n0iFyAeQ2WAbJTOPZRHc/Os439tc3qC/vJvK1rR3F8elY28UgVsYysO0oUh1wE3UFC7e5RuYgey2D\nSCig2UTVEJGLReQxz9eciPymiPyBiJzwHL/Fc5/3i8hBEXlGRF673DUoa5NM1jDjFv20OoBsjClJ\nLa0nm+gzPzjCPz50tK7niLmD2bsC7YkZxFMZzszHc8Fj8IhBC4PI8TKuGREh4JP2WwbJ0vkDjVzh\ne8VgrVsGy57SYIx5BtgDICJ+4ATwVZzJZh82xvyl93wRuQy4Dbgc2AR8R0QuMsas3XdZWRLnokns\ndMJWWwapjMEYCiqQ67EMGrlSjKeyDPXYbCJn9KKdndAKTszEMIZcjQHAcE/73ETFm3DAL+1vR1FU\nCQ2OZTAxV/v1pzJZoslMQczgbAtjLZ2m2W6im4DnjTHVLpVuBb5ojEkYYw7jjMW8rsnrUNYA1kXU\n1xVouRjYuEBhzKB2BXI8lWWxTkGIpxyXRVfAT9a0PrPGBonH+rpyx0IBH+siwda6icrEDMAJInei\nHUXxOnpCgbpaUdtWFJ10E+0fn+H+p8605bmaLQa3AV/w/P4eEdkvIp8WkXXusc3Acc854+4xRSnA\nisGFo73MRFMNDTFvlNz84zLZRNXcRNFkmsU62yF7U0uh9c3qFhLOZmYLpiytrkKO5VpAFG4vAb90\noIV16XjL7pCfaB2tqIvFoJGU1GbxhYeP8b57Hm/LczVNDEQkBLwB+D/uoTuBC3FcSKeAv7Knlrl7\n2b8QEblDRPaJyL7JyclmLVVZJXjFIJ01zNe56S4F6wpqNGYQTWbq2lggPyTeBjRbXXg277ZS6C0n\nBi2MGcSSGfw+yc2StgT8vra3oyhnGdQbQC4Wg56Qn8U2i8F8PE1vuD0zl5tpGbwO+LEx5gyAMeaM\nMSZjjMkCnyDvChoHtnrutwUom45hjLnLGLPXGLN3dHS0iUtVVgO2+vjCMacPfyvTS3OWgXfSmb96\nzCCbdYLOi8l0XVaLncXb7AZ4lbBi0Fe0mbS6JYW1gIrjIUGfdKQdRallECCWytQcWmTFIJ9aGmi7\nZbCYWJ1icDseF5GIbPTc9ibgCffne4HbRCQsIjuB3cDDTVyHskaw1ccXuENZWhk3sBuzN4Dsc69u\nK1kG1h2SNbWv8o0xxN25wOE2WQYLriXV1xUsOG7dRK1yu3kH0HsJ+H1tzyaKp8pbBpDvIluJuZKY\ngZ9kJtvWuMdCIk1PuPS9bAVNkRwRiQCvAd7uOfznIrIHxwV0xN5mjHlSRL4MHADSwLs1k0gpx/Ri\ngv6uQC4dspVVyIlUqZsIqs9B9roaFpOlvumCx3cfo60xg3gav09K8uxH+8LEUhkWk5mWXHU66Zzl\nxEDaX2eQKl2Lt411ccaTl3JiYO830N2eEq2FRIbNg121T2wCTXlFxpioMWbYGDPrOfZWY8yVxpir\njDFvMMac8tz2IWPMhcaYi40x32zGGpS1x9RikuHeMOsiIQBmWmoZlLqJnN99Fd053s28VhDZuhe6\n3HYUznM2JgZHpxZ53937674ynY+n6A0HStw1rS48K+enByebqFmWwQ+eP8sb/vbBmu9hNFlqpdgB\nN7ViPXk3kXN+JDcYp33XrqvVTaQoTWV6MclQTygnBudaGDNIlskmAkcM6rIMamws1iXhtQwSDbqJ\nHnh2ki8+cpyjU4t1nT+fSJdkEgGM9jpXmq0Sg2gqUzD/2NLMbKKHD0+zf3y2Zt5/vIxlkB99WV3A\nZ2MpV7yL5ye3L73UcROpGCjnOVYM+ruDiLTaMrB1BqVuokoxA++mUKvWwNs9M+cmatAysAHhepvM\nLVTIRLGWwcR8vKHnr5d4MlMw/9gS8Pua5iaacIWsmkWWymRJZUyJlVLvgBtv9TEUuonaxUIiXZIN\n1ipUDJQVy9RikuGeEH6fMNAd5FwLW1LYDT8UKLYM/BUtg1gjbqJUvqhtqamlc3Hn9U/VOehnoYJl\nMNzrWFqt6lxaPP/YEmxiOwpr1ViBrLQOoLQC2RWHWu6eUjFo74CbZDpLMp2lt0pco5moGCgrEmMM\n51zLAGBdJFQ1m+j07PKucstVIIO1DMr/88cacRO5G393aOmppXMx1zKoUwzm4+mSTCJw3kuR+h+n\nUaLJdPmYgd/XNDdRPZZBuVkGkK+7mI9Xv7goFoPuNruJ7GtTN5FyXjMXS5POGo8YBCuKwcGJBa7/\nk/t58LmzS36+StlE4YCPZIWr2eJsomp4m7ct1TKwm9d0vW6iCsFHv09YFwkx1aLCMzvRrRgnm6hJ\nlsGcI/4L9YhBkTCN9LozHWqI4WwsXSAGNsWzXQFk+9rUTaSc19g5BtalsS4SqhhAHj/nzPn9/nNL\nr1K3G37ZmEGFTbshN5Enm6grsLTU0lzMYLG+TXw+XtnfPNQTylV4N5tYmdx+aJ5lYIzJVVBXE4N8\nw7zCtdiEhFpuMu8sA4BI0Hkv21WFnBMDtQyU8xm7UQ25XTYHI6GKAWSbAvjwkeklP1/OMgiWppZW\nsgy8V4i1/MjlsomWahnU7yZKlVQfW4Z7Qi11E5WtM/BJUwq2ZqKpXCXzQh0xg+LU0lDAx0B3sKZl\nVMlN1K7Rl4sqBoqS3/CGC9xE5S0DWxz0+Pjskk34XAWyf2mWQbUrVPBaBv4l1xlYy6AeN1EynSWR\nzpYNIINjcbXCTZTN5iutiwn6m9O1dMKTElvNIrOWV7nCsuGeEGeriGE6k2UhUd5N1K4A8rzGDBTF\naxm4YtATIpbKlHWtWMsgnTX85Pi5JT1fIp1FBIL+wgKtcMBfM2bQ3xUgWkMM4p6rVNvmonHLoH43\nUa2ryla5iXIWUKU6gyaklnrrI6qJ8L4jzt/C+v5wyW21xLC4SR2Qc++1y02kloGiUCoGgxHnn7Jc\nEHkunibgE0TgkcNLE4OkO+WsuFrXsQwqZRM5WTN9XUEWGsgmAnIDbhohF0CuYxPPdywtzSYCZ8jN\nTCzV9F5BsQp+eoCAr/GYQSqTLXEPeusjKonBXDzFpx48xKsvXc/24Z6S24d7wlXfx9NugHpDf74V\nhM8ndAf9bXMTLVToOtsqVAyUFcnUQpKIp0BrqEoV8mw0xVBPiIvX9/HIEuMGzsjL0g2saszArW7t\nCdcesJ7zX7suonDAX7U1djGZrGExmcEnjhjU6rg5X2GWgWW4N4QxNL12o5KfHhyrq1E30d//x2Fe\n+Zf/XiBa1k000huqKAaf/Y8jzMXTvPem3WVvdyyDymJwxhWD9QOFfYGcz7rNAWStM1DOZ6YXEzmr\nAJwAMpSvQraBvut2DvHjY+eWdLWbSGdKMomgeszA9r2JhAK1YwapDEG/EHBjEl3ByhZHOexV4uZ1\n3WQNudnQtc6vFEC2722zXUWV0jlhaW6iJ0/OcS6a4sRMLHdscj5BJORnrK+rbMxgPp7ikw8e5tWX\njnHlloGyjzvcG2Y6miRTYT2n3LqVjUVi0M45yLZ2pV1dS1UMlBWJrT62rOuxbqIyloErBi/cMUQ0\nmeHJk3MNP18ilS2pPgb3Cr5KNlEk5Kc3HKidTVTU1rkr6G+oHYWtPt7hujyma8QNauWo21nI9aap\n1kssVd1N1KhlcHTKSRs+dDbfj2liPsFYX5jecKBsBfJnf3CE2ViK9950UcXHHe6xllF5MTw9G8cn\nzuwHL5Fg+0ZfLiSc3kgBf3u2aRUDZUUy7ak+hnxueLl/Xq9lACzJVZRwYwbF2BbW5Xr/27YLkZC/\nZp1BcV/9rmBjAeRiMaiVI5+LGVRKLW1RS4pqlkFwCY3qjk87YnDEKwZzcUb7wvR2BUqK/ebjKT7x\n/cPcdEllqwBqv/5Ts3HG+rpKNuJIW91ErWkxXgkVA2VF4ohB/qosF0Au49aYiztisL6/i+3DER4+\nvFQxKB8zAMrGDaJuq+becOmmVEysqJWyEzOof1Oxm/sOd9BPrRoBm5ZYrh0FtM5NFM3NP64w3KaB\nCuSFRDr3Og97xGByPsFYXxc94UBJncH3nj3LbCzF219+YdXHrmUZnZmLs2GgdI5ApI1zkCtVkLcK\nFQNlxWGMcWcZ5C2DcMC5Aq/kJrKVoi/cMcS+o+canuKVSGdK2lc7z1t5DrJ1E0XC/rp6Ey3HMsiJ\nwXAEqC0GuZhBBTdRq/oTxatkE9mxl/V+NsdcFxGUisGo6yYqzuKy7rMLRksziLyM1GEZeDOJLN3B\nQFtTS9tVYwBNFAMROSIij4vIYyKyzz02JCLfFpHn3O/r3OMiIh8RkYMisl9ErmnWOpTVz2IyQzKd\nLXATgbOBFQeQM1nDfDydE4PrdgwxvZjk+cmFhp6zkpsoJwZlNm7rJuoJB+rqWuq9Wu4K+BtKLbVp\npdtdMahVeDYfTxHwSdnXBK3rTxRLVQsgO2upFLQt5pjrIrpgpIcj7gyHWDLDfCLNWH+Y3rCfhUTh\nxcFMtLQ+oBzDtj9Rhdd/eraaZdCumMHqtgxeaYzZY4zZ6/7+PuB+Y8xu4H73d4DX4cw+3g3cAdzZ\n5HUoqxi70ZWIQU9pszq7Sdp//qu3DQLw+IlZGiGRzhIq4yYKVXETORO9AvSEAiTS2apZTLFUJpdW\nCk6dQSOppdYyGIyEnFYKdQSQ+7pKp5x5aUXhmfWnVyo6A+rOKDo27QjAjReNcuJcjEQ6k6sxGO0N\n0xsOEk8Vvu8zMWe6W7BG0HWwO4ivgmU0H0+xkEiXZBJBm1NLK8yjaBWtdhPdCnzW/fmzwBs9xz9n\nHB4CBkVkY4vXoqwSck3qyljTCGdiAAAgAElEQVQGxW6i4krRLeucK+eTM421tE6kyqeW5tpNl7mK\nz9cZ1G5glijq8b9Uy6CvK1BXX6GFKk3qLMM91XPtl0K8imUQ9Dnvb70ZRcemowx0B3nB1gGyxgkm\n2+rjsf6uXMql932fiaZqWgXgFJAN9YTKTkqz7dDLWQbdwdqZY81iMblK3UQ4g++/JSKPisgd7rH1\ndvax+33MPb4ZOO6577h7TFFKqo8tg2VmGhSLQXfIz7pIkJOevHTLg8+drejOSWYqZxPZ24uJJtOO\nm8jd5Kv21i/KJgoH/Q3HDEIBZwTjcG+otpsokaY3XMtVEmp6amm0Rp0BUHdG0bHpGNuGIuwc6QXg\n0ORiruBsrC+ci4d4azxmY8lcskEtnCrk0tdvq483DnSX3BYJOQWGjcaklkI9gt5MmikGLzXGXIPj\nAnq3iNxY5dxytmvJuysid4jIPhHZNzm59PbEyuoi36SuMMd7XSRYkk1kB754rwY3DXaXiMGZuTj/\n5VM/4ouPHKcciVT1bKLimIFtyNYdzFsG1fLPYyV1Bo0Vnc3F0/S7G8NQT+1NfD6eqhg8trTCTRRL\nZQj5y+fG22P1zjQ4NrXoiIGbTntkapEJd6Me7QvnLTKPGJyLpuoXgwpVyLbgrFwAORL2kzXlEwqa\nzaqNGRhjTrrfJ4CvAtcBZ6z7x/0+4Z4+Dmz13H0LcLLMY95ljNlrjNk7OjrarKUqK5ycZdBbahnM\nxdMFPmJrGfR35/9pNg125/6hLTZP3Zuv7iWRzpbNJqpkGdiCMduOAqjan6i4k2ej7Sjm4in63TTR\noRp9dZy1pCtWH1uGe8Kciza3P5GTQlt+Wwn66rcMMlnD+LkY24YjDESCDPWEOHx2kcmFBAGfMBQJ\n5TZKb+HZTDTJYHeo0sMWMNwbLutus26isTIN7uzIzFa7ilIZp+vsqhMDEekRkT77M/BTwBPAvcDb\n3NPeBnzd/fle4JfcrKLrgVnrTlKU6cUk4YAv536xDLlXfLOeVgzluktuGugqaF8A+cwU+72YRDpT\n0r4avDGDwg3TGyjtsbNxq7VTTpYWnSUz2boza5wRls7zjPSGavYnqitm0GsL+ZrXnyiWLD//GPKW\nQT1icGo2Rjpr2DbkxIB2DEc4fHaRibkEI71hfD7JbZSLBW6iFAN1u4lCnC2TTXRqNs5wT6hsf6X8\nHOTWZhS1e+QlNM8yWA88KCL/CTwM/LMx5l+APwVeIyLPAa9xfwe4DzgEHAQ+AbyrSetQ1gBnFxIM\n94RKMmHW9ZRuXmXFYLCb+Xi6YMbt8XOOOByvKAa1LIPCK0Fvpa39h63WnyieLrxithtNvYVnjtvH\nWgahmv2J5uvIRGlF4ZkTVC//vLY9eD1uIltjYMVg50ivIwZujQHkW23Y990Yw0w0xWAdAWRwxGA+\nni75DCoVnIHjJoLmjL48OLHA06fLt06xr6mWdddMmvJMxphDwAvKHJ8Cbipz3ADvbsZzK2uP6cVk\nLg/ci40hTM4n2DXmBBVnYymCfim46t406AT+Ts3GcxuoFYHxczEyWYPflxcaY4zbwrr+mEHMMzjF\nFlhVch2kMllSGVNoGQTyc5AjdXg15uPpnA87v4knSoLsufMT6YrVx5ZcFe5CAugD4NGj57hofW/N\n+1aiODbiJeCr3zKwFlxeDCLc/eME3cFFLhx1PntrkdmNczGZIZ01DcQMnNd/bjHFhoH8mk/Nxtk8\nWEEMQqUZTEvlD+59kqdPz/Pg77yy5D1bWMWWgaI0jeK+RJaN7j/oqdm8C8j2JfJaEZvc87yuIisG\nyUw2ly1iqTT/GCrHDPJuIl/uCtxrGXx533E++t2DQOFgG0t4SZaB8zz5Tbz8FX0i7RTt1Qog5/rz\nuJbB5HyCn/u7H/DZHxypa03lcGovym8rNpvIppZ+/bET/NE3DpTNzDk2HSXgk1yuv80oOjIVzfny\nc9lEbszAFiTWHzNwzit2FZ2ejbG+TPAYnNRSaI6b6ORsjLMLCb748LGS2/JuovZ0LAUVA2UFMrVQ\n2IrCsmkgf8VvmYsXDi2HvGXgzSg6Nh1lk7uxeNscQD4zpJEKZLsZdAcDRMpkE31l3zgff+B5jDG5\nFNKCCuRg3jKoBydm4LzO4k28mIUaTeosxW6iR49OkzXw9On5utZUjnrcRLbo7N7HTvKpBw/zrQNn\nSs49Oh1ly7ruXJxhx0gkd5vtJFqcTZSrPm4gZgCF72M8leFcNFW24Mx5zua5iWzNxMe/d6jkomC+\nRjuRVqBioKwonL5EiZKCMyhfQzAXy2fZWMb6uvD7hFNu4Vk8lWFiPsFLdo0ApXEDu9FXswyK21jH\nPa2abYaJN5vo1FyMuXiaY9PRsoVYdoRiPYVn6UyWaDLjsQxqiEGivo0k15/IvTK2YyIPTjTWysNL\ncUM+L3k3kfNe2pqBD/7TgZLN9fh0lK1DeQHY4ZlWNupetQf9PsIBX+712vhR3TGDMi0p7FCbDWVq\nDICaLsF6iacyzMfTvGjnEKdm49z96ImC2/OzDFQMlPOUaDJDPJUt6FjqZeNAYQ2BdRN58fuEDf1d\nufPG3eDxi3YO4fdJSUaRvSorHzMoX4HszSby+cQpRnI3pWzWcGbW2WAePzHrmf5V2I7Cee7alsFC\nUQdSG0ivVHhWq321JdefyBWVfUcdMTh0drHuLKdibFV2OfJuIuexJ+bjXLS+lxMzMe7894MF5x6b\njubiBeBsinaW8Vhf/m/DaVZXaBkM1hOEIW9heQPolYbaWLqblE1krYI3X7OZF2wd5GP/frCgMrvd\n849BxUBZYdh/zHJuInDiAV43UTkxsOednC3MILpgtIdNg10lYpC0bqIqXUsrxgzcq+AeTxvrqcVk\n7vzHT8wu2zKwm7stOgv6fVX7E9UabOPFFp7FUxmePDnLWF+YZDrL+LnyWVe1iBWl0Hqx/YLSWSel\n9uxCktdctp5b92zi7x44lKsBmY2lmImmck35LNY6GPWIQY9XDGJuzKBON1FfOEDI7ytoSVGtFQWQ\nS3dermWQr6Tu4tdfuYvxczHufSxfajWvYqCc7+Srj8uLQT2WAdgqZOcf+7i7sW0dirBtKFLGMqji\nJvKXjxl43UTgbBLWtD/tEasnTsyWHfhiA8j1iIF1f3gzfKr1J8qLR+1N0fYn+s/jM6QyhrdcuwVY\nuqvItugoR8BTdDa96IycHOvr4ndvuZSgX3jfPfuZj6dy4u21DCDflrrYMiiJGdTpJhIRtwo5L6rV\nqo8h34Dv9Fx8WS0prGUw2hfmpkvHuGRDH5976Gju9tVcZ6AoTcH+Y5ZLLQUno2gunmYxkSabNcxV\nEIONA92cmo2RzRqOTUXpCvoY7Q2zbShSGjPIiUHpJubzCUG/VMkmylsG1nVgs52u3DzA4+OzuYEv\n4eDSAsjFlgG4V/QV3ES2rXM9V5W2P5F1Ef3CC53GAEsVg3gqW1EMrGWQymRz3UfH+sKs7+/i999w\nOY8cOcctH/k+/7TfuULeWiQG12xbx3BPiLG+/EbtHX05G3PGRFaKWZRjqEhUT8/G6OsKVNyEQ34f\nI70hPv7AIV76p//G733t8Yq1K9WYXMj3WBIRrr9gmINn5nMCs5BIEw74anZfbSYqBsqKopZlsDlX\nQxBjMZkma8pfCW4e7CKVMZxdTHD8XJSt6yKICFuHIkwtJgvSQG08oNwMZHBbRxTXGbhiYN09PaG8\nu8JeXb7msvXMxdM8d8bJzukuakcB9aWW5juWeiyDKk3mctlEDbiJ9h2ZZtdYL9uHexjpDS9JDNKZ\nLMlMtqKbyNvCOt991BH9n9+7lS+//XqyWfj4A4eAUsvgLddu4aHfvangc/KOvmykFYVluDdcYBmc\nnotXjBeAY01867dezl+85Squ3DLAFx8+zp0PPN/QcwJMzsURyWd07RiOsJjM5FxW7e5LBCoGyjL5\n2k9O8Pf/cbhpj2dz5yvFDGwnyZMz8bJ9icqdd2w6lrvK3D7kuBq86aXV3ETgzkEurkBOORXFPtf1\n4e1zf2o2TtAvvOJip5/Ww4edq+7uMqml5YbmFFMuzbBaf6K5OgPIkO9P9OjRc+zdvg6AXWM9HGxw\nOBBUH2wD+WwixzJw3SS9+Y332u1D3PfeG3jT1Zt52a6RksI3ESm5UvaOvpxpoEmdZaTEMohXzCSy\nDPWE+Lm9W/n4W/eyZ+sgB880/l5NulX2NnV2uxsPsTMcFhPt7VgKKgbKMvnsD4/wv39UWjSzVKYX\nE3QFfRVz1e1V26nZWNlWFBZvrcG4JzPFfvfGDZJV3ETOcV/ZOgPvGiOeQObp2RgbBrq4eEMfQb+w\n76gzk7lcO4p4Q5ZB/vmGeyr3J1pIpAn563OXWNGdi6e5NicGvRycWGjYJ54Tg4puonzMoNgysAx0\nB/nwL+zhH3/1RXU9p3f05UwFl2E1ijuXOuMuy7soy7FrrLfhqXpgR3fmhdAGy4+cdf4uF+LpXIV1\nu1AxUJbF4bOLTe3gOLWYLGld7WXDQBcixZZBOTeRIwYHTs4xn0izZZ3zuxUDr583USWbCKxlUOwm\nKnSH9IYCRBN5y2BjfzfhgJ+LN/TlApuFjerKN8ArR94yKHQTVepP1EgffG+l9wt3DAGwa7SX+Xg6\nt2HXSzzpvJbKbqJ8NtHEXJy+rkBD/v1yeEdfzi7BMhjuDRNLZYgm06QyWSYXEjUtAy+7xnqZWkyW\ntFavxaSnxxI4Q5l84hTbgbqJlFXGucUkM9FUzmfbDCpVH1uCficQfHImxlwVy6C/2+kZ9KPDU0Be\nBAYiQfq7AgWWQb7OoFLMoNQyiKUKs2YiYX8uA+SUZ37ulZsHcucUtrC2AeQ6LAM3mOj1lY+4AXYb\niPXSyEZihXekN5S7Ot015vQpajRuEE25VdmVLANfvs5gYj5RkBW0VLyjL2dijccMrBhOzif42Hef\nx5jSWEU1bJ+kRt1qE/OJXCU1OBccmwa7OerOel5QN5Gymjjs/uFGq/Txb5RKfYm8bHTnFVRzE4kI\nmwa7eez4DFCYmbJtOFIkBs5GXymAXM4yiCYLi6t63TqDbNZwejae66N0hSsGIoViE/T78PukbjdR\nsfVz0Xpnw376VGnriHoG21is8F67fV2uv5NtAtjoBhdLVncT5VtYZ10xqByorZfc6MtEZmkxA/f1\nv+vzP+bD33mWN+7ZxM+8oP4JvPa9er4B4cxmDWcXEiUusu3DEY66sazFRHtHXoKKgbIMDk86YpDM\nZOuea1uLqYVEVTcROJlCJ2djZaecedk02J2rdi0Qg6L0UptNVDlm4C/J+ikuroqEAmSN03wsmcmy\n0c1Tv2rzIOBkHRW35O4K+OpKLZ2LpUs29wtGewgFfDx5crbk/HraV1vW93UR9AsvuXAkf6w/TG84\n0LBlUK6ewktxNlG54TGNYt+Xs4sJEuls3X2JLPZv7enT8/zeT1/Kh39hT8W/g3JsHuwmHPA19F7N\nxlKkMqbAMgAniJy3DDJtdxO199mUNcWRqfzUsGgyw0D38q4tnL5E1d1E4GQKfffpSWZjKXxCxUCb\nbUw31BMq+MfaNtTDtw+cybWyrta1FJzc8mS62E2UKbBget0rVLspWL/zRRt6nRbbZa6Ww8FSkSnH\nnGeWgSXo93HJhj4OnCrth7+QSFdNj/QyEAnyL795I9s9YikiXOgGkRuhVjZRMJdNZJiYj5dshkvB\nXj2fcFuONOomunhDH7dft5WfuWpTrndVI/h8wgWjvQ1ZUbbGYLTITbZjOMK5aIrZWIqFRCr3N9Uu\n1DJQlsyhs14xWH7cIJrMkEhnK9YYWDYOdBFLZTg2HaW/O5hL7yzGZhRtXVcYENw2FCGVMblW1tUa\n1YETWC7uIVTsJrKZRXYDtZuxDSJ3lXnsei2Dec/8Yy+Xb+rnyZNzJVk/jVgG4Pi9i2cW7xrNi4Ex\nhn986CjP1OhmGiuqyi7GWgYz0STxVLYploF9nbb/VKNuoq6gnz9581VLEgJLoxlFE3PlxWCbm/Z8\naHKBeCq7+txEIrJVRL4rIk+JyJMi8l73+B+IyAkRecz9usVzn/eLyEEReUZEXrvcNSidwbqJIN9l\nsRzZrKnLjWRT/GrFDOwm//TpuaqphDkxKAoI5tJLXf9sIp3FJ5Qd4g4VLINkJtfbHvJXqM+778lG\nz3CUW1+wmZeW2Wy6gv5cAPm7z0zwW196rGyDuEoxgMs29jMTTZXMe16oY7BNLXaN9TIxn2AunuJP\n/+Vpfu9rT/B3NYqrbFZZ5a6ljhjYORPNiBlYMTgx43yW9XYsbSYXjvYwfi5WVzIAwORCvvrai23T\nba291ZhNlAZ+2xhzKXA98G4Rucy97cPGmD3u130A7m23AZcDNwMfE5H22kPKsjHGcGRqMbexVrMM\n7nzgeV7z1w/UzFu3FbW13UTOJvL85GJ1MXDPqyQGNm6QSGeq+okdd06pm6g7lP/3sYHM5ycWCPiE\nEU/c49duvIC/+LmSQYAFj/s333mOr/7kBPc9XjoKfD6epi9c+jov2+QEp588WegqaiS1tBI2MPo7\nX9nPxx84RDjgY//4TNX7FPdrKkZECPgk11uqKdlEXYWWQaMxg2awa6wXY6jbOvD2JfJi/y4PnFyl\nYmCMOWWM+bH78zzwFLC5yl1uBb5ojEkYYw7jzEG+brnrUNrLxHyCaDLDFZv7gepdHB87PsORqWjN\nvPVc9XHNALJzxZ/JmqpisM1NlbTpf5ZNg12EA77cEJdK848t5SyD4qIzaxkcnFxgfX9XRdeVl3DA\nRzyV4eDEPI8dn0EEPvrdgyWFZM5gm9KN4ZINfYhQEEROpDMkM9llbyRWDL75xGnefPVm3vmKCzl0\ndrFgrnQxtbKJwHEV2QaCzXAT2XiRFYN1dbavbia5jCKPpVyNyXmnsLL4M4qEAoz1hXPivqpTS0Vk\nB3A18CP30HtEZL+IfFpE1rnHNgPHPXcbp7p4KCuQQ+4f/uXu1Wk1y8BegT9zprrP2bZXqOUmGukN\n56pZyxWcWbasi/C1d7+UW/dsKjge8Pu4cvMAjx132kQ4848r/ys4MYO82GWzzvQyb6DUbkrTi8nc\n2M1adAWd+oX/8+g4fp/w/tddwtOn57n/6YncOalMllgqU9bt0xMOsHOkJ3clCeWb2i2Freu6WRcJ\n8rorNvDnb7mKPVsHMQaeOFF+gDt43ERVrKygz8cZtzZitIluIttyu9GYQTPYMdyDT+qvy5hwC86K\ns8vASS99ynUTrbqYgUVEeoG7gd80xswBdwIXAnuAU8Bf2VPL3L2s/0BE7hCRfSKyb3JysllLVZqA\nzSS6bJNjGVSKGRhjcjn9tQKQUzVmGVh8PsnNqK3VpnnP1sGynR/3bB3kiZNzJNNZxzKosoGF/IUB\nZFsbUFB05vm53grWrqCfhUSae358gldePMYvv3QnW4e6+dvvHsx3r7Sbe5n+S+CIsddN1EiTumoE\n/D6+/zuv4mO/eA0Bv4+rtjgpssWuol/85EP80TcOAI6byNuvqRzBgA9jnNqN5QoW5DfMifkEIb+v\nYiZTK+kK+tk6FKm71mCySo3F9uGe3N/aqnMTAYhIEEcIPm+MuQfAGHPGGJMxxmSBT5B3BY0DWz13\n3wKcpAzGmLuMMXuNMXtHR0ebsVSlSRw+u0go4GOX64KpZBmcXUjmrhifrWEZTC0k6A76K/Yl8mLn\nITfai8ayZ9sgyXSWp07NkUhnKhacQWk2kXWHFBedWepN6+wK+Hnq9JwziH7vFgJ+H+98+S7+8/gM\nDx48C5RvReHlso39nJiJMeu2vMgNtikTY2iU3nAgd/U61BNiy7pu9o/nXVLHp6P8x8Ep/uGHR5mY\niztxlBqbsQ0ij1W4Mm6UkFuZbYwTL2jGYy6FC0frzyiaLKo+9uJN8V11YiDOu/8p4CljzF97jnvL\n+N4EPOH+fC9wm4iERWQnsBt4eLnrUNrLoclFdgxHcr7sSpaB7cIY9AvP1OjuOF1HjYHFZussVQyu\n3uZ4LR87PkMiVcNN5MYM7NV68ZQzcNpRWCoNRil53KCziQ33hHjVJWMA/Oy1m9nQ38X/ut+xDubK\nNKnzcrlrmT15ytmknzjhfK/lalsKL9gyyP4Tecvgu8847qxkJstnfnCEaJUpZxZrpTUjeGzpczfN\nTmQSWXaN9dY9LnSiqC+Rl+0j+VnPq04MgJcCbwVeVZRG+uci8riI7AdeCfwWgDHmSeDLwAHgX4B3\nG2Oa189AaQtHphbZOdKTu4qvZBlYF5Ed3uENji4m0jzuudI8u5isWWNgsWmjSxWDTQNdjPaFeez4\nDMlMrZiBs8HZ4rRy3TnDAX8ujlF3zMB1Tb3x6s25TTIc8POeV+3i4SPT/PF9T9UUA+umO3ByjtlY\nir/81jNcs22Qq7cO1rWGRrhyywDHp2O52M79T02wc6SHW67cwD8+dJSzC4mqwWPI1xo0I63UYl1F\nnYgXWHaN9pJMZ2sOukmkM8zGUpXFYDVbBsaYB40xYoy5yptGaox5qzHmSvf4G4wxpzz3+ZAx5kJj\nzMXGmG8udw1Ke8m408N2jDgtEQI+qZhNdHQqigi86pIxFpOZXI45wMf+/SC3fvTB3D/Q9GKi7ita\nmza6VDEQEfZsHfRYBtVjBpBvdV3OTeT87vzz1h8zcB735/ZuKTj+iy/axttevJ1PfP8wf/Od54DK\nsZGR3jDr+8McODnHX3/rGaYXk3zw1ivqymZqlKu2OMkC+8dnWEyk+eHzU7zqkjHuuPFC5uJpvvfs\nZG0xsG6iJmQSWeymOdBg9XEzuXDMuaKv5Sqyw2sqWUZ2zjOs4gCycv5wcsbpv3OBa9JGQv6KYnBs\nOsqG/q7cRuKNG3z36UmyBr70iJNcNr2QrDjuspjlWgYAV28b5PDZRU7PxaumltrbbNwg7yYq/Ge1\nm1K9MYNbrtzIe165i0s29BccFxF+/2cu581Xb+ZHh51ZCNUaz12+aYAHnp3kHx46yluv355rjtds\nbAfWx8dn+Y+DZ0lmstx0yRh7tg5y3c4hsqZyKwpLK9xEvSvCMqiv02ulGgPLQCTIQHeQkN9XNY7V\nClQMlIaxbSh2jjjB4x7PUPJijk05g2V2u102bXrp5HyCA6fmCPiEL+87TjqTbchN9LLdI/zOzZdw\n3c6hJb+OPa4r5dh0NHf1X44Sy6BCq+ZIyO8UnNUpaC+6YJj/57UXl73N5xP+/C1X8ZrL1hPwCeuq\nvC+XbexnajHJukiI//ZT5R+vGfR1BblgtIf/HJ/l356eoC8cYK87A+HtN14AQHeN4H8r3EQ2c6qT\nMYOBSJCNA13c//RE1eLKiTmbVlv5b2THcKTtNQagYqAsgcOuKWzL52tZBtuGIvR3Of8sz7rppd9/\nzkkVfvcrdzExn+Cf9p8kmc7WHUAOB/y88xUXLuvq6aotg9jkk3CVK9piyyDmDnEpdhP1hAOs7+/C\n3yQXTcDv485fvIb7f/vlVVNoX+CK2vtvuXRZllI9vGDLIPvHZ/i3pye48aLR3Pv/yovHuGJzf64g\nsBJ29GW1zbBRVkLMAOBdr9zFw4en+af9hVXksWQmdyFRqUmdl0s29DfVcqoX7VqqNMyRqSi94UAu\nPa4nHCg74CaWzDAxn8gNTblofV8uo+j7z51luCfEu1+5iy8+coyPftfpezNUo/q4mfSGA1w01scz\nZ+arBpBDfjeAnHMTuZZBkYBsHuxu+oYU8Pty83ErcdMlY9zzrpe0JGhczJWbB/jqT04A5DKgwLFk\nvvKOl5St6fBig+zNFINczKAD1cde/q/rtvGlR47xoX8+wKsuGcu1Af8vn/wRfV0B/uFXXpRzE1Wz\nHj/w+kubOiOkXtQyUBrmqVNz7BzpyeV0V7IMjrtVobY30MUb+nh+coFUJsv3n5vkZbtHCAV8/Pze\nrTlfa72WQbO4epuzgVbNJgpYy8B5jZVm/f75W67if91+dSuWWRWfT7hm27q25Ni/YGt+WM8rLi6s\n/ekK+mtaRdYyaG4A2fkcOukmAvD7hD+69QrOzCX4yP3P8fTpOW6764ekMllOzcb52Tt/wI+PzTDU\nE6oqmv1dwdykvHaiYqA0xOnZOI8cmS7YCCKhQNnUUtsV1Dbgumh9H8l0lm8+cZqzC0lu3O08xs/v\n3Zpz19QbM2gWNm5QNZsoUF82UU84sOxuoSudyzYO4PcJV28drDvY7yXgF3xSu/9UI9gCu067icCp\nX7nthVv59IOHue2uh/D7hC+9/cV84deuJ5bK8L1nJ5syx6EVqBgoDfG1x06QNfDma/LpkJGQv6xZ\na4d7WzfHxW4Q+ZPfPwTADRc5bZ23DkW4wRWGVhRLVWOPaxlUrUAOlM8mqtaDZ63SHfLz3pt28+uv\n2r2k+wf9PkZ6w02Lq0C+Y2yjg21axX+/+RJ6uwL0hAJ8+e0vZtdYL1duGeDLb38xmwa62DlS3e3X\nKTRmoNSNMYa7Hx3n2u3rCv6ge0LlYwbHp53Ywjr3im3XWC8isH98lks39hdklLz3pl1Egv66q3eb\nxe6xPl6wZSBXvFWOYsugnh48a5nfuGlpQgCwc8Rp6tZMnKZvzY1DLIehnhDffO8NREKBgoD+rrFe\nvvPbL8fXoZYZtVAxUHjKTfG06Z+VePzELM9NLPDHb7qy4HgkXMEycOcdWF92d8jP9qEIR6ai3HhR\n4bCXa7cPce1bl54mulT8PuHr73lZ1XOsC8nGDJwpZ/qvsxT+v9dfVvukBrnlyo1cMNLbET97JTZW\nKDxcyX83K3dlSluYiSZ588d+QCyV4ZINffzMCzbxCy/cWjbb4e5HxwkFfPz0VRsLjlvLwBhTEMQ8\nNh1l91ihwFy0vs8Rg92rp/FgqIybqBPdMZXyBP0+rtzSmkK78wmNGZzn/O+HjxFLZfiNm3YTCfn5\ni399hjd/7Aecmo0VnJdMZ7n3P0/ymsvWl+Syd4f8ZA0FnT2zWcPxc7FcWqll7451jPSG2LtjHauF\n4phBPJWp2XZBUVYbKgcHiTYAAA04SURBVAbnMcl0ls/+4Ag37B7hv73mIu5510u5510v4dxiktvv\neogzc/nZuv/29ATnoinecs2WksfpcTdGb3rpmfk4yXS2ZOTkr7zsAh74f19ZNXtnpREuihk4U85W\nz/oVpR5UDM5j/vnxk5yZS/DLL9uZO3bNtnV85pevY3I+we2feIivP3aC//GNA/yPfz7AaF+YG3aX\nDnaPhG0b63wQ+WhRWqnF75O2N+BaLvmYQd5NVGnou6KsVlQMzlOMMXzqwcPsGuvl5UX++2u3r+Pv\n/+t1nJqJ894vPsbnHjrKhv4u/vhNVxIoUyzTk2tjnbcMjuXSSiMl5682ymUTqWWgrDVW1yWasiSy\nWcPDR6a5/6kzbB/u4Weu2sRTp+d44sQcf/LmK8umSF63c4jv/PbLOTuf4NKN/VXz8O1gF2966fHp\nKH6f5LqLrmZCngrko1OLPHV6nl/Yu7XGvRRldaFisIZJprP8zf3P8pVHxzkzlyDgE9JZwwe/cYDh\nnhBDPSHedPXmivffPNhds/EYQMR1mcQ8lsGJczE29HfV7FWzGvD7hIBPSKSz/OE/HSDoE97zql2d\nXpaiNJWOiYGI3Az8DeAHPmmM+dNOraWdpDJZTs/GSwKrjTIxF+fzPzrGUE+ISzb0ccmGfgY85fix\nZIZ3fv5R/v2ZSV596Rgf+OnNvPrSMQ5NLvKVR8f5xv5TvPPlFzbF991TJmZwdjHJSJv7DLWScMDH\ndw6c4bmJBT5wy6Wsb3NxnKK0mo6IgYj4gY8CrwHGgUdE5F5jzIFOrKddPHp0mt+95wmeOTPPbS/c\nyvtedwmDDXZaNMbwpUeO86H7nsoNS7fcsHuEt714B3t3rOPXPrePfUfP8SdvvpLbr9uWO+eKzQNc\nsXmAP3jD5U15TZDv0eONGZydT9Q95GU1EA76eW5igd1jvfzfL93R6eUoStPplGVwHXDQGHMIQES+\nCNyKMxe5aRhjePzELNOLSc5Fk0wvpvCJcyUbCfk5t5jk+clFDp9dpCvo47KNA1y6sY8NA134RAj4\nhXTGEE1mWEymc0PRswZ84mSZhIM+BGExkWYxmSaRzhLwCQG/D78IIiDA956b5AsPH2fTQBe3X7eV\nL+8b5ztPneG3XnMR24d66Ar6EIHTswlOzsQ4F006PU02D7BtOMJzZxZ49Og5vrH/JI8cOcf1Fwzx\nx2+6kp5wgKdOzfHjo+f48r5xfvVz+wj5fRgM/+v2q3n9VZua+ZaWJWcZeGIGU4sJrthcucXDasMO\nuPngrVesCdeXohTTKTHYDBz3/D4OvKjZTyIi/PzHf0g8la14TiTkZ+dID7Fkhm8dOEOVIUXLwu8T\nfu2Gnfzmqy+iJxzgv1y/nd+953E+8NUnyp7vE8iWWcumgS7+7GevdDt9OoHf9f1dvOLiMX79pt18\n+8AZvv7YCX7xRdu58aL2VPnmLAO3JUU2a5haSNY98Ws1cMFoDy+/aJQXXzjc6aUoSkvolBiU69RU\nsvWJyB3AHQDbtm0ruUM9fPyte+kN+1kXcQKmxsCCexU/2B1ifX84t6lGk2mePj3P9EKSjDFkswaf\nT+gJBYiE/YT8Pvw+wSdCJmtIpDPEU1kMht5wgJ5wgJDfRyZrSGcNmazBuC9rXSRU4Ge+fNMA97zr\npTx1ao5oMkMinSGTNazv72LTYDc9IT/PTy7yxIlZjk4tsnt9H9dsX8emga6KfeuDfh+3XLmRW67c\nWPb2VmH7rVjLYC6eIp01S2pxvFL5/K82/VpFUVYUnRKDccCbm7cFOFl8kjHmLuAugL179y7pmv3l\nZa6OK82TjYQCXLOtfW0S/D6pOrz84g19XLyhevO4lYDfJ4QDvlw20dkFO81p7QSQ2zE4RlE6Saec\nn48Au0Vkp4iEgNuAezu0FqUJeEdfnl1IAtVH+ymKsrLoiGVgjEmLyHuAf8VJLf20MebJTqxFaQ7e\nATd5y0DFQFFWCx2rMzDG3Afc16nnV5qLd8DNlGsZtHuesaIoS0dz5JSmEAn7c3UGUwsJfOIEzRVF\nWR2oGChNIRLKi8HkQpKhnlBT59wqitJaVAyUphAJBXLtKKYWEhovUJRVhoqB0hR6PJbB2YWExgsU\nZZWhYqA0hUg4QNQGkBfXVvWxopwPqBgoTaEn5GfRppbOJxjuUTFQlNWEioHSFLpDAWKpDNFkmsVk\nhpE+dRMpympCxUBpCj1us7rj0zEARtQyUJRVhYqB0hQibhtrO/tYLQNFWV2oGChNwVoGR6cWATRm\noCirDBUDpSnYNtbHc5aBioGirCZUDJSm0BN2LAPrJhqu0CZcUZSViYqB0hTstLNj01H6wgG6gv4O\nr0hRlEZQMVCaQs5NdC6m1ceKsgpRMVCaQo8rBsl0VquPFWUVomKgNIVIOO8WUstAUVYfyxIDEfkL\nEXlaRPaLyFdFZNA9vkNEYiLymPv1d577XCsij4vIQRH5iOhw2TWBtQxAJ5wpympkuZbBt4ErjDFX\nAc8C7/fc9rwxZo/79Q7P8TuBO4Dd7tfNy1yDsgLoCvqwsj6sYqAoq45liYEx5lvGmLT760PAlmrn\ni8hGoN8Y80NjjAE+B7xxOWtQVgYiQsTNIBpVN5GirDqaGTP4ZeCbnt93ishPROQBEbnBPbYZGPec\nM+4eU9YAtiWFWgaKsvoI1DpBRL4DbChz0weMMV93z/kAkAY+7952CthmjJkSkWuBr4nI5UC5+ICp\n8tx34LiU2LZtW62lKh2mJ+RnEi04U5TVSE0xMMa8utrtIvI24PXATa7rB2NMAki4Pz8qIs8DF+FY\nAl5X0hbgZJXnvgu4C2Dv3r0VRUNZGdhaA21FoSirj+VmE90M/A7wBmNM1HN8VET87s8X4ASKDxlj\nTgHzInK9m0X0S8DXl7MGZeVgq5C1fbWirD5qWgY1+FsgDHzbzRB9yM0cuhH4oIikgQzwDmPMtHuf\ndwKfAbpxYgzfLH5QZXUSCQcI+oX+7uX+WSmK0m6W9V9rjNlV4fjdwN0VbtsHXLGc51VWJj0hP8M9\nYbR0RFFWH3oJpzSNt754Ozddur7Ty1AUZQmoGChN4yUXjnR6CYqiLBHtTaQoiqKoGCiKoigqBoqi\nKAoqBoqiKAoqBoqiKAoqBoqiKAoqBoqiKAoqBoqiKAogbqPRFY+ITAJHl3j3EeBsE5ezGtDXvPY5\n314v6GtulO3GmNF6Tlw1YrAcRGSfMWZvp9fRTvQ1r33Ot9cL+ppbibqJFEVRFBUDRVEU5fwRg7s6\nvYAOoK957XO+vV7Q19wyzouYgaIoilKd88UyUBRFUaqwpsVARG4WkWdE5KCIvK/T62kFIrJVRL4r\nIk+JyJMi8l73+JCIfFtEnnO/r+v0WpuNiPhF5Cci8g33950i8iP3NX9JREKdXmMzEZFBEfmKiDzt\nft4vXuufs4j8lvt3/YSIfEFEutba5ywinxaRCRF5wnOs7OcqDh9x97T9InJNs9axZsVARPzAR4HX\nAZcBt4vIZZ1dVUtIA79tjLkUuB54t/s63wfcb4zZDdzv/r7WeC/wlOf3PwM+7L7mc8CvdGRVreNv\ngH8xxlwCvADnta/Zz1lENgO/Aew1xlwB+IHbWHuf82eAm4uOVfpcXwfsdr/uAO5s1iLWrBgA1wEH\njTGHjDFJ4IvArR1eU9MxxpwyxvzY/XkeZ4PYjPNaP+ue9lngjZ1ZYWsQkS3ATwOfdH8X4FXAV9xT\n1tRrFpF+4EbgUwDGmKQxZoY1/jnjTGPsFpEAEAFOscY+Z2PM94DposOVPtdbgc8Zh4eAQRHZ2Ix1\nrGUx2Awc9/w+7h5bs4jIDuBq4EfAemPMKXAEAxjr3Mpawv8E/juQdX8fBmaMMWn397X2eV8ATAJ/\n77rGPikiPazhz9kYcwL4S+AYjgjMAo+ytj9nS6XPtWX72loWAylzbM2mTolIL3A38JvGmLlOr6eV\niMjrgQljzKPew2VOXUufdwC4BrjTGHM1sMgacgmVw/WT3wrsBDYBPThukmLW0udci5b9na9lMRgH\ntnp+3wKc7NBaWoqIBHGE4PPGmHvcw2es+eh+n+jU+lrAS4E3iMgRHPffq3AshUHXnQBr7/MeB8aN\nMT9yf/8Kjjis5c/51cBhY8ykMSYF3AO8hLX9OVsqfa4t29fWshg8Aux2Mw9COIGnezu8pqbj+so/\nBTxljPlrz033Am9zf34b8PV2r61VGGPeb4zZYozZgfO5/psx5heB7wJvcU9ba6/5NHBcRC52D90E\nHGANf8447qHrRSTi/p3b17xmP2cPlT7Xe4FfcrOKrgdmrTtp2Rhj1uwXcAvwLPA88IFOr6dFr/Fl\nOGbifuAx9+sWHB/6/cBz7vehTq+1Ra//FcA33J8vAB4GDgL/Bwh3en1Nfq17gH3uZ/01YN1a/5yB\nPwSeBp4A/gEIr7XPGfgCTkwkhXPl/yuVPlccN9FH3T3tcZxMq6asQyuQFUVRlDXtJlIURVHqRMVA\nURRFUTFQFEVRVAwURVEUVAwURVEUVAwURVEUVAwURVEUVAwURVEU4P8HHY4Dzi0ZeaoAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118591f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(all_reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们画出 reward 的曲线，可以发现奖励在不断变多，说明我们的 agent 学得越来越好，同时我们也可以实实在在地看到 agent 玩得怎么样，gym 提供了可视化的过程，但是 notebook 里面没有办法显示，我们可以使用运行 `dqn.py` 来看到 agent 玩的可视化视频。\n",
    "\n",
    "另外，我们这里只使用了简单的多层神经网络来作为 dqn 的网络结构，网络的输入是杆的位置信息和角度等等，我们当然可以使用更加一般的输入，比如说每个状态都是一个图片的输入，那么这种方式更具有一般性，实现上几乎是一模一样，只需要改一改网络结构，同时 gym 中也可以得到每个屏幕的输出，具体可以看看 pytorch 的[官方例子](http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html#)。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
